Cosmos AnomalyGen: package runtime, packaging tool, workflow-block productionization - #2897
Cosmos AnomalyGen: package runtime, packaging tool, workflow-block productionization#2897probicheaux wants to merge 4 commits into
Conversation
- CosmosAnomalyGen model class (cosmos-anomalygen, new image-generation task, custom backend): mask-conditioned defect inpainting whose generate() mirrors the upstream SDG generation-entry contract (anomaly_type, guidance, num_steps, seed, crop-and-paste). Runtime ships inside the model package via the same import_class_from_file seam as the Cosmos 3 Edge generator. - Workflow block roboflow_core/cosmos_anomalygen@v1: clean image + segmentation-prediction placement mask (rasterized like the stability_ai inpainting block) + anomaly type -> generated IMAGE_KIND output. Loads in-process via AutoModel (sam2_video block precedent); local execution only - remote raises, with HARD restrictions declared. - Mocked unit tests for model class and block; changelog. Weight packaging note: the AnomalyGen runtime (paidf-anomalygen / cosmos-predict2 stack) must ship inside the model package; licensing is NVIDIA evaluation/model licenses - review before hosting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ults - cosmos_anomalygen_runtime.py: reference package runtime driving the GA paidf-anomalygen stack (config assembled from the package's frozen ag_config.yaml with base-tower paths repointed into the package; generation reuses the SDG dataset/condition/inpaint path 1:1) - pull_anomalygen_weights.py: assembles the flat package (base towers + trained adapter + runtime + class_names.txt + inference_config.json) with hardlinks and an optional --gcs-dest mirror - defaults now follow NVIDIA's production recipe: guidance 1.5, crop_ratio 4.0 (both overridable); crop_ratio/poisson_blend exposed on the workflow block, which now also reports a visibility output (mean abs pixel change inside the placement mask) so callers can filter empty generations - REMOTE execution now rejected at block construction (compile time) instead of run(); integration test (env-gated), docs page + nav + index row Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qKkor5AsgasrsFz3j7TA2
Validated end-to-end inside the GA container on the real Corning tube+hole checkpoint: package assembled by pull_anomalygen_weights.py, runtime load 27s, generation 10s/image at 35 steps, visibility 49 (>=15 threshold). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qKkor5AsgasrsFz3j7TA2
|
👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely. 🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits. Warning 💸 The Claude reviewer bills in credits, not vibesAutomated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared. Draft when unsure, Ready when you mean it:
However you get there, arrive prepared:
Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.
|
|
🤖 Claude review started at commit New commits are not auto-reviewed. Add the |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| return ["cosmos-anomalygen"] | ||
|
|
||
| def discover_dependent_resources(self) -> Optional[List[DependentResource]]: | ||
| return [roboflow_platform_model(model_id=self.model_version)] |
There was a problem hiding this comment.
Medium/High — declaring a model-manager dependency for a model this block loads via AutoModel, contradicting the sam2_video/sam3_video policy the block claims to follow.
This block loads its weights with AutoModel.from_pretrained(...) in _resolve_model() (in-process, not through the model manager) — exactly like segment_anything2_video/v1.py. Those blocks deliberately do not implement discover_dependent_resources():
# segment_anything2_video/v1.py
# `discover_dependent_resources()` deliberately not implemented: this
# block loads its weights via AutoModel.from_pretrained, not the model
# manager — dependencies stay undeclared (None) for now.and are listed in NON_MODEL_MANAGER_LOADERS_ALLOWLIST in tests/workflows/unit_tests/core_steps/test_dependent_resources.py.
By returning roboflow_platform_model(model_id=...) here (defaulting to required_action=EXECUTION → execution_location=ENVIRONMENT_DEFINED), this block opts into model-manager pre-loading. Concrete failure path: an InferencePipeline initialized with workflows_dependencies_pre_init=["roboflow_platform_model"] under LOCAL step execution → _is_locally_executed_platform_model() returns True (EXECUTION + ENVIRONMENT_DEFINED + LOCAL) → _pre_load_roboflow_platform_models() calls model_manager.add_model(model_id="cosmos-anomalygen", ...) (inference/core/workflows/execution_engine/v1/core.py:191). But cosmos-anomalygen is served only through the inference_models AutoModel path (custom backend, generate() contract) — it is not a model-manager model, so add_model registers the wrong thing / fails, and the block never uses that registration at runtime anyway (it re-loads via AutoModel).
Note CI is green here: test_every_block_with_resource_kind_fields_declares_dependencies only checks that a roboflow_model_id-field block either overrides discover_dependent_resources() or is allowlisted — so declaring the override silences the guard without making the pre-load path correct.
Recommend matching the cited precedent: drop this override (leave dependencies undeclared / None) and add roboflow_core/cosmos_anomalygen@v1 to NON_MODEL_MANAGER_LOADERS_ALLOWLIST. If instead the intent is genuine model-manager execution, the block must actually run through the model manager rather than AutoModel.
Reviewed at HEAD: 456946e
Review summarySkills: review-workflows-blocks, review-inference-models-pkg, review-topic-local-vs-remote-execution, review-topic-input-boundary-security, review-topic-external-contract-and-silent-fallback, review-topic-backward-compat-and-versioning, review-topic-test-hygiene New Cosmos AnomalyGen surface: Blocking finding (inline):
Version / changelog notice (maintainers): functional Tests to add (behavior-level):
Minor doubts (non-blocking): Reviewed at HEAD: 456946e |
|
@probicheaux — a couple of the human-blocked items you called out are release-gating for this block and need confirmation before sign-off. ⏳ This PR is on hold pending your answers — the review will not advance to sign-off until the IMPORTANT questions below are answered (and the blocking finding on IMPORTANT questions
(The NVIDIA license review for the paidf-anomalygen-derived packaging — your item #3 — is a maintainer/legal gate, not a code question; flagging it here only so it is not lost. The docs page correctly marks it "review pending".) New commits are NOT auto-reviewed — add the Reviewed at HEAD: 456946e |
…ackage The GA model __init__ prefetches it unconditionally (early-stop nn metric), so package load fails without it; the earlier smoke masked this because the webapp container had it at the cwd-relative path. The runtime now repoints model.config.correspondence_backbone into the package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qKkor5AsgasrsFz3j7TA2
1 participant
Successor to #2676 (closed when its base stack #2675 squash-merged); the branch was rebuilt on latest
mainas the anomalygen delta only."image-generation"forcosmos-anomalygenmust be what/models/v1/external/statreturns — needs the models-service (GOAT) enum addition; confirm with Inference Core. (On the platform,image-generationis also a billing-meter name in a different namespace — do not reuse it as a meter.)cosmos_anomalygen_runtime.py); it must be registeredtrusted_sourceserver-side, or callers needALLOW_INFERENCE_MODELS_UNTRUSTED_PACKAGES=True.development/cosmos3/pull_anomalygen_weights.pyproduces the package +--gcs-destmirror.What this adds
cosmos_anomalygen_runtime.py(reference,inference_models/development/cosmos3/) — the self-contained runtime that ships inside the package. Builds the GA config (predict2_anomaly_gen_ddp_2b+ the run's frozenag_config.yaml), repoints every frozen-tower path into the package (DiT, VAE tokenizer, t5-large, NV-DINOv2, guardrail dir), and reuses NVIDIA's SDG entry path 1:1 for generation (temp one-line JSONL →AnomalyInpaintDataset→inpaint_image), so preprocessing/RePaint/crop-paste stay byte-compatible withsynthetic_dataset_generation. Requires thepaidf-anomalygen:gacontainer environment (lazy imports; clear error otherwise).pull_anomalygen_weights.py— assembles the flat package (base towers + trained adapter + runtime +class_names.txt+inference_config.json) with hardlinks, optional--gcs-dest.crop_ratio4.0, crop-and-paste on, Poisson off — model class + workflow block.crop_ratio/poisson_blend; newvisibilityoutput (mean abs pixel change inside the placement mask, 0–255) so callers can filter/regenerate empty generations (≥15 worked in the Corning CV study); REMOTE now rejected at construction (compile time);discover_dependent_resources()declared.COSMOS_ANOMALYGEN_PACKAGE_DIR).inference_models/docs/models/cosmos-anomalygen.md+ nav ("Image Generation" group) + index rows; changelog.Test evidence
tube+holecheckpoint @ iter 35k): package assembled by the tool → runtimeload()27 s →generate()10.1 s/image at 35 steps, output at native input resolution, visibility 49.3 (≥15 threshold), PSNR 12.8.inference_modelsunit suite: 1552 passed (4 pre-existingruntime_introspectionfailures reproduce on pristinemainin the same env).test_dependent_resources) fixed by this PR.black/isort/flake8(CI-pinned versions) clean oncheck_dirs.🤖 Generated with Claude Code
https://claude.ai/code/session_015qKkor5AsgasrsFz3j7TA2